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1. INTRODUCTION 

Most of the applied problems derived from the following branches: engineering, biological, 
mathematics, physical, chemical, and the rest of the scientific branches are non-linear. Researchers have 
continued to develop numerical methods that solve this type of problem as in [1], [2]. In this article, we will 
discuss how to solve a system of nonlinear equations, which we can represent by: 


F(x) =0 (1) 


So, we know the function F: R” —> R” is a nonlinear function. The premise of this function R” refers 
to the real space of the dimension-n measured by the Euclidean standard ||-||. As for the methods that can 
solve such a non-linear system, they are the iterative methods, such as the Newton method, Quasi-Newton 
method [3]-[8], and the derivative-free method [9]-[11]. When solving in (1) using the most commonly used 
iterative method through the linear sequence as in [12], as the search direction dx is obtained from (2). 


F(x) +I (Xx) dy = 0 (2) 
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Where J(x,) is the jacobian matrix that equal to F’(x,) or an approximation of it. One of the good 
characteristics of Newton’s method is the speed in reaching the optimal solution and its rapid convergence, 
but it requires the computation of the Jacobian matrix. The first idea of the double-search direction suggested 
in [13] relies on generating duplicates from the (3). 


pe 2 
Xk+1 = Xk t Agee FARCE (3) 


Where a; > 0 is the step-size, x;,41 is the new point of (3), x; is the previous point while e, and c, are the 
first and the second search directions, respectively. In this article, we will shed light on approximating a 
matrix J (xx) using a diagonal matrix (I is the identity matrix) i.e., that (4). 


TX) ~ Wel (4) 


We can define f(x) as a modular function that defines it (5). 


1 
f(x) = +F (5) 
Note that the problem of equations in (1) is equivalent to the following global optimization problem (6). 


min f(x), x E R” (6) 


In the double search direction method (3), the iterative information is used in multiple steps, and 
curves are searched to generate new iterative points. Researchers continue to develop a special type of search 
direction, for example, Petrovic and Stanimirovic [14] deal with a double-direction to solve unconstrained 
optimization issues. The transformation of the double-step length scheme is suggested in [15], [16] to boost 
the numerical efficiency and global convergence properties of double-direction methods. It is also possible to 
compute the step-length alpha, either by using exact or inexact line searches. Thus, inexact line search [17], 
[18] is the most commonly used approach in this field. A fundamental requirement of line search is to 
minimize function values properly, i.e. to evaluate the function values (7). 


IF eal S IF Goel (7) 

We organized the article in the following order: section 2, deals with the two new algorithms (S-RA 
and D-RA). Section 3 deals with introducing some new theorems that prove the convergence of the newly 
proposed algorithms (S-RA and D-RA). Section 4, concerns the numerical results which demonstrate the 


efficiency of the newly proposed algorithms when compared to the standard (HWY) algorithm. Section 5 
deals with general conclusions. 


2. TWO NEW ALGORITHMS (S-RA AND D-RA) 

In this section, we suggest reducing the two vector directions (3) into a single that with relying on 
the projection technique to find that direction of research. This is made possible by allowing the two 
directions to be identical, i.e. e, = Cg. We propose that the eg and cp in (3), the unique search direction is 
described as (8) 


Cx = Ce = -Pk F (Xx) (8) 
Now, put (8) into (3), we get (9). 


Xr+1 = Xk + Akl Pr (1+ ay) )F (Xe) (9) 


Through (9), we can conclude that the double-direction will become (10) 
dy = pr (1 + ay) F (Xx) (10) 


Set the acceleration parameter used in (10) as (11), 
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[Isiell” lle] 


Waar =e Ferl (11) 


where Yg = Fk+1 — Fk and the difference between the two-point is Sk = Xg}1 — Xp . Therefore, through the 
(9) and (10), we can get (12). 


Xk+1 = Xg + kdk (12) 


The projection approach relies on the use of a monotone case F to accelerate and change the new 
point using repetition. As in the (13). 


Zk = Xk + andy (13) 
The hyperplane, as an original iterative, is (14). 
Hy, = {x E R|F (2y)" (Xx — 2x) = 0} (14) 


To start using the projection technique, we use the update of the new point Xg+1 as given in the [19], 
[20] to be the projection of x, onto the hyperplane Hg. So, can be evaluated as: 


Xg+1 = PolXk — SkF (Zx)] (15) 
= EEO" Akzo) 
kT Fax) II? (16) 


In the next paragraph, we will present the standard (HWY) algorithm [12] and the newly proposed 
algorithms which are divided into two parts: first of one (S-RA) algorithm which uses (12), (8), and (11) and 
the second (D-RA) algorithm which uses (12), (10), and (11). To clarify the idea of the numerical algorithms 
used in this research, we present the steps of each of these algorithms in detail. 


2.1. Algorithm (HWY) [12] 
Input: Given xo, Yo E (0,1), a > 0, € = 1074, w, and w, > 0, set k=0. 
— Compute Fk = F (xp). 
— Test the stopping criterion. If yes, then stop; otherwise, continue to the next step. 
— Compute search direction dx using (10). 
— Compute step length a; using this line-search: 


f (Xk + edi) — f Œk) S =w: llag Fell? — We llar dell? + ne f Ce) 


= Set Xk+1 = Xk + aay. 
— Compute F(x,41). 
2 
— Determine w,4, using Wea = ae 
kok 


— Set k=k+1, and go to 2. 


2.2. New single search direction algorithm (S-RA) 
Input: Given xo E Q, Wo, r,0,u €(0,1), a > 0, € > 0, set k=0. 
— Compute F, = F (xx) and test If ||F,|| < € yes, then stop; otherwise, continue to the next step. 
— Compute search direction dẹ (using (8)). 
— Set Zg from (13) and compute step length a, using this line-search: 


=F (xq + UTE dy) dy, 2 OMT Fe + u rk AM Mell? (17) 


— Ifz, EQ and ||F (zx) || < £ stop, else compute x;,,, from (12). 
— Determine w,,, using (11). 
— Set k=k+1, and go to 2. 
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2.3. New double search direction algorithm (D-RA) 
Input: Given xo E Q, Yo, r, o, u €(0,1), a > 0, € > 0, set k=0. 
— Compute F, = F (xp) and test If ||F, || < € yes, then stop; otherwise, continue to the next step. 
— Compute search direction dx (using (10)). 
— Set Zg from (13) and compute step length a, using this line-search from (17). 
— Ifz, EQ and ||F(z;) || < £ stop, else compute x;,,, from (12). 
— Determine y,,, using (11). 
— Set k=k+1, and go to 2. 


3. CONVERGENCE ANALYSIS 

In the previous section, we proposed two new algorithms (S-RA and D-RA) depending on the 
parameter wW,,,,. Now in this section, we will present an affinity analysis for the second algorithm, which is 
more general than the first as in the coming theorems, but before that, we must give the basic assumptions a 
space of attention which is: 


3.1. Assumption A 

Assumption A means that the special solution of (1) in N stands for x*. Since F’(x,) is 
approximated by w;/ along the direction s,, we might mention another assumption of the same idea. 
— Suppose there is a set level defined by: 


2 = {xl FÆI < FoI 


— There is an x* that belongs to R”, where F(x*) = 0 is true. 

— Let the function F be differentiable and continuous in some neighborhood, that is, N of x* contained in 
N. 

— On N, i.e., there is a Jacobian of F restricted and positive definite, i.e. there are a positive constants M > 
m > 0 are such that: 


IF'ODI <M, YxE N. (18) 
And 
m|ld||? < dTF'(x)d, Vx E N,d E R”. (19) 


3.2. Assumption B 
If we consider that Yp I is a good approximation of F'(x,), which means that: 


WCF) — We Ddell < ell FOI (20) 
where € € (0,1) is a small quantity [6]. 
3.3. Theorem (descent direction) 

Suppose assumption B holds and that new algorithm (S-RA) and (D-RA) produces {xx}. Then, dp in 
(8) in the direction of the descent of f (xp) at x, i.e. 

Vf (x,) dy < 0 (21) 
Proof: We will divide the proof into two parts, each part concerned with an algorithm to change the search 


direction in each of them as follows: 
Part 1: when dealing with the first algorithm, we will need a search direction from (8), as: 


Vf (Xp) dy = F (xp) F' (xy) dy = F(x)" [(F' (Xn) — We Dy — F(x) 
Vf (xn) de = F(xK)" (F'(X) — Pr Dede — WF ll (22) 


by Cauchy-Schwarz inequality, we have: 
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Vix) de < IFICE Oe) — x Deel — WF CI? (23) 
If (20) satisfy then, 
Vf (xi) de < ellFx)II? — IFO? s -A- E E) (24) 


Hence for € € (0,1), this proves of part | is true. 
Part 2: when dealing with the second algorithm, we will need a search direction from (8) and (10), as: 


Vf (xn) dy = F(X)" F' (xn) de = F (tu) F(X) — We Ddy — (1 + a IF Dll (25) 


by Cauchy-Schwarz inequality, we have: 


Vi (XK) de SM FONG Ce) — Pr Ddell — (1 + a IF OI? 
If (20) satisfy then, 


Vi (x) de < EFEN- A+ @ IF Oa? S -aA S- € + aE El? 
Hence this proves part 2. This means that the two new proposed algorithms have descent search directions. 
We can deduce from the theorem (descent direction) that the norm function f (xx) is a decline for dg, which 
implies that ||F (x41) 11 < IE @Ðll <...< [|F(@o) ||. This implies that x, E€ Q. 
3.4. Lemma (bounded Yk+1) 

Suppose that assumption A holds and {x,} is generated by an algorithm (S-RA) and (D-RA). Then 


there exists a constants M > m > O such that for all k: 


[lsll?-llyell?] — M-m? (26) 


Fea)? T m? 


Proof: 
From assumption A we get: 


Yk Sk = m lIsxll? (27) 


from [18], we have: 











M? yk Sk = myx? (28) 

then, 
1-M? 

[m llskll? — m Ilyell?] < [yf sk — M? YE Sie] > Ullsull? — llyel?] < yts, 
from the theorem we have: 

m |Isxll < [Frail < I| Fk+1 — Fell < M Ilsell (29) 
hence, 

-M2 
[liseliye] < L Tee [1—-M2|m a 
Fes? T m?lskll? 7 m? 


The inequality (26) is true. Using (26), Yk+1 is generated by the update of (11) and we can deduce that p,44/ 
inherit the positive definiteness of w,J. 


3.5. Lemma (bounded dx) 
Suppose that assumption A and B holds and {x} is generated by an algorithm (S-RA) and (D-RA). 
Then there exists a constant b>0 such that Vk > 0, 





An effective new iterative CG-method to solve unconstrained non-linear ... (Rana Z. Al-Kawaz) 


1852 O ISSN: 1693-6930 
dill < b; (31) 

where i=1,2. 

Proof: We will present two parts in this lemma, each of which depends on the search direction resulting from 


an algorithm as in: 
Part 1: From (8), (11), and assumption A we have: 


lall = |] -—2, (32) 


2 2 
Ilseell -val 
e lF &@k+o)ll 


and using the result of Lemma (bounded w;4,), 


—M2|M 


[1-M?] 
dll sem? Fell S [BillF@od II] s b (33) 





where B,>0 and b, = B,||F(Xo)|l- 
Part 2: From (10), (11), and assumption A we have: 


(1+a,)F (Xx) 
ldel = |- =A (34) 
lsxll“ -lyzl 


Z 
e FoR 


and using the result of Lemma (bounded Wk41), 


[1-M?|M 1-M?|M 
larll SA +aje mm [FGI S (FGI + ellF ODE ™ ] (35) 
1-M?|M 
dell < [FŒ] + Be "7 ] < b (36) 


1-M?|M 
where B,>0 and b, = (||F(%9)|| + B,)e m? .The following theorem deals with the global convergence 
property. To prove that under a few suitable conditions, there exist an accumulation point of x, which is a 
solution to the problem (1). 
3.6. Theorem (global convergence) 


Suppose that assumption B holds, {xx} is generated by an algorithm (S-RA) and (D-RA). Assume 
further Vk > 0, 


[F(T ag] 
a, > rE al 7 
k Z Ta (37) 


where T is some positive constant. Then 

Lira ||F oll = 0 (38) 
Proof: From (31), and (Descent Direction Theorem) we have: 

Lim lisk = 0 (39) 
and the bounded of ||d, ||, we have: 

Lim alld ll? = 0 (40) 


From (37) and (40) it follows that: 
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lim |F (xx)" diel = 0 (41) 


In this stage of the proof, we will take two parts according to the two new algorithms as in: 
Part 1: According to the (S-RA) algorithm and from (8), we have: 


F(x)" de = Py IF Cal? => [ee [LF On) del = WF N? 
and as we imposed in the theorem: 


lbr [Eaelldell? > WF CRIP (42) 


2 2 
[sesal oea] hmm 
While |p| =e Feo <e m? <6 





so, from the (42), then 
6 
0 -zalde = F(x)? = 0 (43) 


therefore, in (42) is true and the proof for part | is completed. 
Part 2: According to the (D-RA) algorithm and using (10), we have: 


F(x)" de = =p (1 + a IF end? 

IF II? = |v. F(x)" dell = lla. F(x.) I? < [Wi Fe) dzl (44) 
while Wi | < 6 as in part 1, so from the (50), then 

0 © SEX dil = FEl = 0 (45) 


therefore, the (38) is true and the proof for part 2 is completed. 


4. NUMERICAL PERFORMANCE 

In this section, we will present our numerical results for comparisons between the two new proposed 
algorithms (S-RA) and (D-RA) and the standard (HWY) algorithm which is devoid of the derivative to solve 
certain nonlinear test problems. In our implementing all three algorithms, we used the Matlab R2018b 
program in a laptop calculator with its Corei5 specifications. As for the tools used in the two algorithms, they 
are as follows: Yọ = 0.6 and 1,r = 0.9,0 = 0.02, u = 1,w, = w, = 1074, ||F(x;)|| < 1078. The program 
finds the results on several non-derivative functions through several two initial points indicated in the 
Tables 1 and 2. 


Table 1. The initial points 








Name of Variable Initial point 
xy (1,1,1,..,1)7 
Xp (0.2, 0.2, 0.2,..,0.2)7 
X3 (20, 20, 20,.., 20)" 
X4 (rand, rand, rand, .., rand)? 





These algorithms we implemented within dimensions n (1000, 2000, 5000, 7000, 12000). All such 
algorithms are recognized by their performance in (Iter) the number of iterations, (Eval-F) the number of 
evaluations of functions, (Time) in second CPU time, (Norm) approximation solution norm. The 
test problems F(x) = (fi, fo, fas ---, fy)’ where x = (x1, X2, X3, Xn)", for i = 1,2,...,n and Q = RÌ are 
from [21]-[24] and listed as shown in Table 2. Using Dolan and Mor’e style [25], the Figures 1-3 are used for 
comparison between the (HWY) with (S-RA) and (D-RA) algorithms when switching the search direction. 
The Figures 1-3 are about the initial point 1 because it is the best performance. 
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Table 2. Define the problems 








No. Problems 
1 F;(x) = x; — sin xi 
2 F(x) =e*i-1 
3 F(x) = Ve (x, -1),i = 2,3,.. n- 1. 
n 


1 
= ` 2 — -5 
RO= 7 xj —1/4,c=1+*10 


J=1 


4 Xi 
F(x) =l + 1) — = 
5 F,(x) = min(min(|x;|, x7), max(|x;|, x7) 
cos(X1+xX2) 
B= e mt 
cos(xipatxitxi1) 4 
Fix) =x, -e n+ , fori = 2,3,..,n— 1 
COS(Xn—-1+Xn) 
F(x) = Xn e nH 
7 i 
F;(x)= —e*i — 1 
n 
8 F,(x) = e1-1 
Fix) = e= ram 
9 n 
F(x) = Dbl! 
10 


neo = Ye 


i=1 
11 F;(x) = max |x;| 
i=1,..n 


n 
F(x) = X [eil eZine 


i=1 
n 
F(x) =) la 
i=1 
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Figure 1. Performance of iterations for the (S-RA and D-RA vs. HWY) algorithms: (a) S-RA and HWY and 
(b) D-RA and HWY 
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Figure 2. Performance of function evaluations for the (S-RA and D-RA vs. HWY) algorithms: (a) S-RA and 
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Figure 3. Performance of time for the (S-RA and D-RA vs. HWY) algorithms: (a) S-RA and HWY and 
(b) D-RA and HWY 


CONCLUSIONS 
The results, presented in the six figures show the efficiency of the two new algorithms (S-RA) and 


-RA) when compared with the previous standard (HWY) algorithm, and their efficiency is better by taking 


the first initial point and increase when increasing the dimensions in the variables used. The new algorithms 
have given a clear convergence in reaching the optimal point for solving non-linear functions. 
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